System for text assisted telephony

ABSTRACT

An arrangement for providing captioned telephone service is provided which permits persons who are hard of hearing to receive captioning of their telephone communications as they need it. A personal interpreter/captioned telephone device can dial a relay on a second telephone line while the assisted user if conversing with a hearing user over a first telephone line. The second telephone line connects to a relay which provides voice to text translation service and returns a text stream to the captioned telephone device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of Ser. No. 09/938,194, filed Aug.23, 2001, now U.S. Pat. No. 6,603,835, which is a continuation-in-partof application Ser. No. 09/783,679, filed Feb. 14, 2001, now U.S. Pat.No. 6,594,346, which is a continuation-in-part of application Ser. No.09/288,420, filed Apr. 8, 1999, now U.S. Pat. No. 6,233,314, which is acontinuation of application Ser. No. 08/925,558 filed Sep. 8, 1997, nowU.S. Pat. No. 5,909,482.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

BACKGROUND OF THE INVENTION

The present invention relates to the general field of telephonecommunications. In more particular, the invention relates to systems toassist telephone communications by those persons who are deaf, hard ofhearing, or otherwise have impaired hearing capability.

Most modern human communications in both social and businessenvironments takes place through sound communications. Yet within modernsociety there are many persons who have attenuated hearing capability.To assist those persons in making use of our telephonic communicationsystem built for the hearing majority, there has been developed a systemof telephone communication which has been principally used by the deafcommunity. That system makes use of a category of device known variouslyas a telecommunication device for the deaf (TDD), text telephone (TT) orteletype (TTY). Current TDDs are electronic devices consisting of a keyboard and a display as well as a specific type of modem, to acousticallyor directly couple to the telephone line. Modern TDDs permit the user totype characters into their keyboard, with the character strings thenencoded and transmitted over the telephone line to be displayed on thedisplay of a communicating or remote TDD device.

Most TDD communication is conducted in an idiosyncratic code specific tothe community of TDD users. This code, known as Baudot, evolvedhistorically at a time when many telecommunication devices for the deafwere based on mechanical or electromechanical devices rather than thecurrent technology based on digital electronic components. Accordingly,the Baudot protocol was constructed for a set of constraints which areno longer relevant to present date devices. The original Baudot protocolwas a unidirectional or simplex system of communication conducted at45.5 Baud. The conventional Baudot character set was a character setconsisting of 5 bit characters and the system encodes the bits of thosecharacters in a two-tonal system based on carrier tones of 1400 and 1800Hertz.

The system of TDD communications is widely used and in fact has becomeindispensable to the deaf community throughout the industrialized world.Deaf persons extensively communicate with their neighbors and with otherdeaf and hearing people remotely, using the TDD system. In addition,systems have been developed to facilitate the exchange of communicationbetween the deaf community and hearing users who do not have access toor utilize a TDD device. In the United States, telephone companies haveset up a service referred to as a “relay.” A relay, as the term is usedherein, refers to a system of voice to TDD communication in which anoperator, referred to as a “call assistant,” serves as a humanintermediary between a hearing user and a deaf person. Normally the callassistant wears a headset that communicates by voice with the hearinguser and also has access to a TDD device which can communicate to thedeaf user using a TDD appropriate protocol. In normal relay operationsin the prior art, the call assistant types at a TDD keyboard the wordswhich are voiced to her by the hearing user and then voices to thehearing user the words that the call assistant sees upon the display ofhis or her TDD. The call assistant serves, in essence, as aninterpreting intermediary between the deaf person and the hearing personto translate from voice to digital electronic forms of communication.

A system to assist users of the telephone system who are hard of hearingbut not deaf has been described. This system, sometimes referred to astext enhanced telephony, makes use of the existence of relays tosupplement telephone communications for users who can hear, but haveattenuated hearing capabilities. This systems includes, in its simplestembodiment, a visually readable display connected in series between thetelephone used by the assisted user and the connection to the telephonenetwork. The text enhanced telephone call is connected through a relaywhich transmits both the voice of the hearing user at the other end anda text stream of the words spoken by that user on the same telephoneline. The details of the concept and of some embodiments of that systemare disclosed in U.S. Pat. No. 6,075,842, the disclosure of which ishereby incorporated by reference.

BRIEF SUMMARY OF THE INVENTION

The present invention is summarized in that a relay system to facilitatethe translation of information and communication between deaf andhearing persons includes a call assistant who re-voices the words of thehearing person which are spoken to the call assistant. The words spokenby the call assistant are recognized by a speech recognition computerprogram which has been trained to the voice pattern of the callassistant, such that the words are promptly translated into a high speeddigital communication protocol. That high speed digital communicationmessage is then transmitted electronically promptly by telephone to avisual display accessible to the deaf person.

It is an advantage of the invention described herein that the callassistant does not have to type most, if any, of the words spoken by thehearing person in the communication session so that the overall speed ofcommunications from the hearing person to the deaf person isdramatically increased.

It is an object of the present invention that the design and utilizationof a relay operated in accordance with the protocols described hereinpermits the introduction of small hand-held personal interpreter whichwill enable on the spot communications between deaf persons and hearingpersons wherever the deaf persons might go.

Other objects, advantages and features of the present invention willbecome apparent from the following specification when taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a schematic block diagram of a re-voicing relay.

FIG. 2 is an exterior view of a personal interpreter enabled by therelay of FIG. 1.

FIG. 3 is a schematic block diagram of the personal interpreter of FIG.2.

FIG. 4 illustrates the operation of a captioned telephone servicesupported by a relay.

FIG. 5 is a schematic illustration of the concept of the two wirecaptioned telephone device connection with sound inter-connection.

FIG. 6 is a schematic illustration of the connection of a personalinterpreter/captioned telephone device connected to two telephone lines.

DETAILED DESCRIPTION OF THE INVENTION

The present invention is intended to provide an alternative arrangementfor the delivery of text assisted telephone services, also calledcaptioned telephone, to a telephone system user with diminished hearing.The present system is intended to take advantage of two developments inthe field of assisted telephone services, the personal interpreter andthe re-voicing relay. The arrangement for using these capabilities hereis designed not to provide text services for the profoundly deaf, but toprovide a text assistance service for those who can hear, but who mayhave diminished hearing. This service is designed to supplement ratherthan replace the transmission of the spoken voice word to the assisteduser. To assist understanding what is contemplated here, the basictechnologies of the personal interpreter and the re-voicing relay willbe described here first, after which the description will return to themain topic, text assisted telephone services or, as they are also known,captioned telephone services.

The personal interpreter is intended to be a small portable devicecapable of delivering, with the support of a relay, text to a deaf userof any human conversation occurring in the presence of the personalinterpreter. The personal interpreter is set up to connecttelephonically to a relay, transmit spoken words to the relay, and thendisplay for the assisted user the text of the words transmitted over thetelephone connection to the relay. To make the personal interpreter workmore seamlessly to produce more conversation-like communication, afaster relay methodology was needed. To fill this need, the re-voicingrelay was designed.

The re-voicing relay is based upon the underlying technology of usingvoice recognition software, operated by a call assistant (a “CA”), toassist in the voice to text translation inherent in providing relayservices. The re-voicing strategy is described in U.S. Pat. No.5,909,482, the disclosure of which is hereby incorporated by reference.The re-voicing relay has its first implementation in providing voice totext transcription services for the deaf. However, the relay voice totext transcription service has use for users other than those who aredeaf. It is envisioned that there are a number of hearing or partiallyhearing users who would have reason to benefit from voice to texttranscription services. Relay voice to text service might also be usefulfor any application in which it is desired to supplement voicecommunications by a text transcription of the voice spoken on thetelephone. Thus, while the operation of the relay will sometimes bedescribed here by referring to an “assisted user,” who may be deaf orhard of hearing, but who also may be a normally hearing person whosimply wants text assistance for some reason. The user at the other endof the line will be referred to here as the hearing user, simply for thepurpose of having something to call that person, even though both usersmay be hearing.

Personal Interpreter

Shown in FIG. 2 is an illustration of what a personal interpreter 10 canlook like. This would be a small hand held device typically the size ofa small hardbound book. It would have a keyboard of minimal size, butuseable by a deaf person who can type. It would have a two or four linedisplay, but the display could be any size that conveniently fits in thecase of the device. The device would also have a key or switch whichwould initiate its operation.

Shown in FIG. 2 is a schematic block diagram of the internal mechanicsof the personal interpreter. The personal interpreter keyboard shown at12 and its display as shown at 14. Inside the interpreter itself is amicroprocessor shown at 16. Not shown, but included within the personalinterpreter, would be the appropriate memory and interface devices so asto allow the microprocessor to be programmed and to operate the personalinterpreter and perform its functions, in a manner well known in theart. Also inside of the personal interpreter is a modem 18. The modem 18is preferably a modem specifically designed for interface with the deaftelecommunications system. Most telecommunications with the deafcommunity are conducted using a Baudot type code. One useful alternativeis for the modem be designed to use the enhanced form of Baudotcommunication known as “Turbo Code” (Ultratec), which is generallydescribed in U.S. Pat. No. 5,432,837, U.S. Pat. Nos. 5,517,548, and5,327,479, the disclosure of which is hereby incorporated by reference.Another alternative is that the modem use a new variant of Turbo Code,one which uses higher carrier frequencies (in the range of 3000–3500hertz) and a faster baud rate (over 100 baud). The most preferredalternative is for the modem to use a digital communication protocolwhich can both transmit and receive digital packets which contain eitheror both of voice and text. The output of the modem is preferably wiredto a cellular telephone 20 included within the case of the personalinterpreter 10. The cellular telephone 20 has a suitable antennaprovided on it so that it may dial a cellular telephone network by radiofrequency communications of the type normally conducted by cellulartelephones. The personal interpreter also includes jack 28 to connect toa conventional wired or land-line telephone line as well. The personalinterpreter also includes a microphone 22 and a speaker 24. A filter 26connects the speaker 24 and the microphone 22 to the telephone 20.

A brief description of the operation and functionality of the personalinterpreter reveals the dramatic improvement and convenience andportability that this device gives to text assisted people. A assisteduser could go into an establishment, be it a government office or retailfacility, in which there are only hearing persons. The assisted personwould carry with him or her the personal interpreter 10. The assisteduser would then place the personal interpreter 10 upon a counter orother surface, open it up, and press the initiation key or start button.The microprocessor 16 and modem 18 of the personal interpreter thenpower up and act in many ways like a normal TDD device operating intelecommunication standard, such as Turbo code. However, there is onecritical difference. The start or initiation key further causes themicroprocessor 16 of the personal interpreter to dial a relay to set upa relay communication session and includes in its communication with therelay a message, using the enhanced command features available inadvanced telecommunication protocols, such as Turbo Code, to initiate aspecial format of relay call adapted for the personal interpreter. Othercodes which permit command functions, such as ASCII or CC ITT, couldalso be used. The first operation is to activate the cellular telephoneand direct the cellular telephone to dial the number of a relayoperating in accordance with the method of the present invention. Thecellular telephone dials the relay. Obviously, no wired connection isrequired to allow the cellular telephone function to establish atelephone connection with the remote relay, but alternatively the jack28 to a conventional telephone line could be used. In addition, when therelay answers the telephone connection, the microprocessor 168 of thepersonal interpreter 10 is instructed to provide command codes to theremote relay. These command codes, a feature possible through the use ofTurbo Code, permits the personal interpreter to tell the relay that thisis a personal interpreter-type relay communication session. All of thiscan happen in the time necessary to initiate the cellular call, perhapstwo to ten seconds.

Then, the assisted person can use the personal interpreter to translatewords spoken by hearing persons in the presence of the personalinterpreter into visually readable text. This is done by the personalinterpreter 10 through an unseen relay. Words spoken by the hearingpersons in the presence of the personal interpreter 10 are picked up bythe microphone 22. Those words are then transmitted through the cellulartelephone 20 to the remote relay. The relay, operating as will bedescribed below, then immediately transmits back, in enhanced TurboCode, a digital communication stream translating the words that werejust spoken. The words are received by the modem 18, and themicroprocessor 16 in the personal interpreter 10, and it is displayedpromptly upon the display screen 14. If the assisted person can speak,he or she may then answer the hearing person with a spoken voice, or,the assisted person may alternatively type upon the keyboard 12. If theassisted user types on the keyboard 12, the personal interpretertransmits the communication by digital communication to the relay. Thecall assistant at the relay then reads and speaks the words typed by theassisted user which are transmitted to the speaker 22 contained in thepersonal interpreter into a voice communication which can be understoodby the hearing users. The filter 26 filters out the digitalcommunication frequencies from the sound generated by the speaker 22.Thus, in essence, the assisted person has a personal interpreteravailable to him or her at all times of the day or night wherever theassisted person is within the range of the cellular telephone system.Also, because the relay is preferably operating in accordance with thefast translation methodology described below, a very conversation-likefeel can occur in the communication session between the assisted userand the hearing persons in the presence of the personal interpreter 10.In order for this communication session to be satisfactory to thehearing users as well as the assisted person, however, the relay mustoperate exceedingly rapidly. It is, in part, to meet the need for theexceeding rapidity of this conversational style of communication thatthe re-voicing relay protocol has been designed.

Re-voicing Relay

Shown in FIG. 1 is a relay intended to provide the voice to textcapability to support the personal interpreter, and which can alsosupport other voice to text services to provide services for assistedusers. FIG. 1 is intended to show, in schematic fashion, how such arelay system can be set up. Shown at 32 is a telephone of a hearingperson. Instead of a telephone of a hearing person, the input could alsobe the microphone of the personal interpreter translator 10 shown inFIGS. 2 and 3. The telephone of the hearing person 32 is connectedthrough a telephone line 34 to a voice input buffer 36 at the relay. Thetelephone line 34 can be an actual physical land line, or two pairbetween the telephones, or can be a cellular or other over-the-airtelephone linkage. The voice input buffer 36 is a simple buffer toensure temporary capture of the voice in the event that the callassistant gets behind and needs to buffer or delay the voice of thehearing person. In any event, the output of the input voice buffer 36 isprovided to a headset 40 where earphones 38 produce the sound of theremote speaking person in the ear of the call assistant. The callassistant is wearing the headset 40 and sitting at a computer 42 capableof communicating in an enhanced Baudot communication, such as Turbo Codeor whatever other code protocol is being used. However, typically thecall assistant does not type the words which the call assistant hears inhis or her earphone 38. Instead, the call assistant then speaks thewords which he or she hears in the earphones 38 into a microphone 39 inthe headset 40. The microphone 39 on the headset 40 is connected totransmit the voice of the call assistant to the computer 42 at which thecall assistant sits.

The computer 42 has been provided with a voice recognition softwarepackage which can recognize the spoken voice of the call assistant andimmediately translate words spoken in that voice into a digital textcommunication stream. It is a limitation of currently available speechrecognition software that the software must be trained or adapted to aparticular user, before it can accurately transcribe what words the userspeaks. Accordingly, it is envisioned here that the call assistantoperates at a computer terminal which contains a copy of a voicerecognition software package which is specifically trained to the voiceof that particular call assistant. It is also important that the voicerecognition system be capable of transcribing the words of the voice ofthe call assistant at the speed of a normal human communication. It hasbeen found that a recently available commercial voice recognitionpackage from Dragon Systems, known as “Naturally Speaking,” is a voicerecognition software which will accomplish this objective and which willtranslate to digital text spoken words of a user at the normal speeds ofhuman communication in conversation when operating on conventionalmodern personal computers. A voice recognition software system known as“Via Voice” from IBM provides similar functionality.

The computer terminal 42 of the call assistant then translates the textcreated by the voice recognition software to a modem 46 out through atelephone line 48 back to the display 50 located adjacent to theassisted person. The display 50 can be a conventional TDD located at thehome of the remote assisted user, or can be the display 14 of thepersonal interpreter 10, or can be any other display or text capturedevice used by an assisted user.

For reasons that will become apparent, there is also a connection fromthe microphone 39 of the headset 40 of the call assistant to theincoming telephone line 34 through a switch 52. The switch 52 canphysically be an electrical switch located between the microphone 39 andthe telephone lines 34 and the computer 42 or, as an alternative, it canbe a software switch operating in the computer 42 which passes the voiceof the call assistant through to the telephone lines as voice, or not,under conditions which are selected by the call assistant, by choices heor she makes at the keyboard 44 of the computer 42. The switch 52 isfunctionally a single pole double throw switch although, of course, ifthis function is performed by the computer it will be a logical not aphysical switch. In the simplest embodiment, the switch 52 is a simplesingle pole dual throw foot switch readily accessible to the callassistant which passes the voice of the call assistant from themicrophone either out onto the telephone line 34 or to the computer 42.

It is a further enhancement to the operation of the relay constructed inaccordance with the present invention that the earphones 38 have noiseattenuating capability. Noise canceling earphones are commerciallyavailable today or, for this purpose, the computer 42 can be providedwith noise canceling sound generation software which would create soundtransmitted to the earphone 38 so as to cancel the sounds of the callassistant's own voice. The noise attenuation or cancellation avoidsdistracting the call assistant, since he or she would then be lessdistracted by the words that he or she has spoken, and thus would beless likely to be distracted from the concentration of the task ofre-voicing the sounds of the voice heard in the call assistant's ear.

Similarly, another option which would be advantageous is that thesoftware providing for the creation of the digital text string by voicerecognition be buffered in its output flow to the modem 46. Before thecomputer 42 would pass the data on to the modem 46, the data would firstbe displayed on the computer screen of the computer 42 for review by thecall assistant. The purpose of this option would be to permit the callassistant to use the keyboard to type or correct hard-to-spell words, orto create corrections of any misinterpretations created by the voicerecognition software, from the words spoken by the call assistant. It isanticipated that if such an option is utilized, it would require fairlyinfrequent use of the keyboard by the call assistant, since frequent usewould clearly slow down the through-put of the communications.

The relay of FIG. 1 can operate with normal TDDs or with a personalinterpreter as shown in FIGS. 2 and 3. In either event, the hearingperson speaks in the telephone 32 and the words are transmitted throughthe telephone line 34 to the voice buffer 36. The voice buffer 36, againoperating under the control of the call assistant, would buffer thevoice signals from the hearing user as needed for the call assistant tokeep up. The call assistant would hear the voice of the hearing userthrough the earpiece 38 and then would re-voice those same words intothe microphone 39. The words that the user speaks into the microphone 39would be fed to the computer 42 where the voice recognition software,trained to the voice of the call assistant, would translate those wordsinto a digital text stream. The digital text stream would be turned intoa digital communication stream by the modem 46 and passed on thetelephone line 48 to a display 50 which can be observed by the assisteduser. Experience has shown that using currently available technology thedelay between the time the hearing user speaks into the telephone 32 andthe time the words appear on the display 50 of the assisted user is amodest number of seconds.

In the reverse, when a non-speaking assisted user types onto his or hertelecommunication device, the digital signals are transmitted to thecomputer 42 which displays them for the call assistant who then voicesthose words into the microphone 39 which words are then transmitted ontothe telephone line 34. Note that the presence of the switch 52 istherefore important in this mode. Since the voice of the call assistantserves two different functions in the operation of this system, thesignal on the call assistant's voice must be switched so that thehearing user 32 only hears the voice for the communications which areintended to be directed to that person. The switch 52 allows for thevoice of the call assistant only to be directed to the hearing person atthe appropriate times.

Note that if the relay of FIG. 1 is used to facilitate a translationbased on a personal interpreter such as that shown in FIGS. 2 and 3,there will be only one telephone line between the personal interpreterand the call assistant. In essence, in a modification of FIG. 1, thetelephone 32 and the display 50 would both be within the personalinterpreter 10. There would be only one telephone line, a cellular link,between the personal interpreter 10 and the call assistant. Notetherefore that the voice of the call assistant and the digitalcommunications created by the computer 42 would then travel on that sametelephone linkage to and from the personal interpreter 10. It istherefore important for this embodiment that the personal interpreter 10have appropriate filtering (i.e. the filter 26) to filter out thedigital communication carrier frequencies of the digital communicationsprotocol, so that they are not heard by hearing listeners in thepresence of the personal interpreter 10. The telephone line must stillcarry voice signals, however, so that the spoken words articulated bythe call assistant in response to digital instructions from the deafuser can be properly broadcast by the speaker contained within thepersonal interpreter.

The provision for filtering of the digital frequencies can be done inany number of ways with two being the principal preferred methodologies.If Turbo Baudot communications are conducted at the conventional Baudotfrequencies of 1400 and 1800 Hertz, the personal interpreter 10 could beprovided with notch filters 26 to filter out signals at those particularfrequencies. It has been found that such notch filters still permit thetransmission of audible and understandable human speech, even if theyfilter at those particular frequencies. As an alternative, it ispossible to change the Baudot frequencies to those which are muchhigher, such as frequencies of 3000 to 3500 Hertz. If this alternativeis selected, the personal interpreter 10 is then provided with a lowpass filter which permits low frequency sounds to go to the speaker tobe broadcast into the environment of the personal interpreter, whilehigh frequencies are excluded.

It is also specifically envisioned that the filter of textcommunications signals from the voice signal can be done digitally orlogically rather than by analog filter. For example, it is possible toimplement the modem at site of the assisted user using a digital signalprocessor, or DSP, integrated circuit. Such a DSP chip can be programmedto recognize to separate voice signals from text communications signalsand to pass along only the voice signals to the speaker at the locationof the assisted user. It is also contemplated that the telephone linkagebetween the relay and the assisted user could be a digital connectionrather than an analog telephone line. For example, the linkage betweenthe relay and the assisted user could be a digital wireless linkage oran internet protocol linkage, wired or wireless. In either of theseinstances, the appropriate protocol calls for the transmission ofpackets of digital data, the packets being marked to indicate the typeof information carried in the packets. The packets might contain, forexample, voice signals which have been digitized or might contain simpledigital information representing the text string of a conversation. Inthe instance of such a digital packet type communication protocol, thefiltering out of the digital text information could be implementedsimply by only converting the packets marked as voice back into sound.Packets marked as containing digital information would be used torecover the text information for display to the assisted user.

Captioned Telephone

As mentioned earlier, captioned telephone is a technology intended toprovide text assistance to assisted user during the course of a voicetelephone communication session. As will become apparent from thediscussion below, a captioned telephone connection can be a single lineconnection or a multiple line connection. There are advantages anddisadvantages of each approach. A single line connection is known in theart, as exemplified by U.S. Pat. No. 6,075,842, mentioned previously.The advantages and details of the use of a multiple line connection aredescribed below. The discussion of captioned telephone will begin with asingle line connection.

Shown in FIG. 4 is an illustration of how a typical telephone callinvolving a single line captioned telephone would be set up. The hearinguser at telephone 62 communicates through a telephone line 64 with therelay, indicated at 66. The relay, a re-voicing relay, communicatesthrough a telephone line 68 with the assisted user. At the site of theassisted user is a telephone 70 used by the assisted user and also acaptioned telephone device 72. The telephone 70 is conventional. Thecaptioned telephone device 72 is constructed to accomplish twoobjectives. One objective is to filter, or separate, the digital signalscarrying the text information from the voice signal. The other objectiveis to take the digital signals and create a visual display of the textinformation for the assisted user. This device is thus intended toassist the user to understand a greater portion of the conversation byproviding a visually readable transcription of the text of the telephoneconversation so that the assisted user can read any words that he or shecannot hear properly. While the captioned telephone device 72 of FIG. 4is illustrated as a separate stand-alone device from the telephone 70,those of skill in the art can readily appreciate that the two functionscan be combined in a special capability telephone station. Such aspecial purpose station can both provide conventional telephone serviceand also include a display which can be used to provide captioning foran assisted user.

The two line captioned telephone is indicated in FIG. 5. In its simplestembodiment, the two-line captioned telephone can be conceptually thoughtof as a personal interpreter used to provide text assistance to atelephone user. Thus in FIG. 5, the telephone of the hearing user 62 andthe telephone of the assisted user 70 are connected directly by a simplesingle telephone connection, indicated by the telephone line 64. Theassisted user then uses a personal interpreter/captioned telephone(“PICT”) device 74 to connect to a relay 76 over a second telephone lineor connection, indicated at 78. There must be some form of communicationbetween the telephone of the assisted user 70 and the PICT device 74, sothat the voice of the hearing user can be transferred to the secondtelephone line 78 to the relay 76. At its simplest, the connection canbe a simple acoustic connection, such as placing the sound pick-upmicrophone of the PICT device in such a location that the sound of thehearing person's voice is picked up and transmitted to the relay. Amicrophone connected to the PICT device 74 could be placed on thehandset of the assisted user's telephone 70. However, it is preferredthat the PICT device be electronically connected to the telephone of theassisted user so that the voice of the hearing user can be transmittedto the relay call assistant with minimal interference or noise. The PICTdevice includes a visually readable display and suitable decodingelectronics and software so that the device can receive and display atext stream received from the relay over the second telephone line.

The main advantage of the two-line approach to captioned telephone isthat the captioned telephone service can be added to a telephone callalready in progress. By contrast, to use single line captioned telephonefor an incoming call, the call must directed through the relay to theassisted user at the beginning of the call. For a two-line captionedtelephone call, however, the call can be started as a normal telephonecommunication session, without the relay or the captioning. Then, if theassisted user decides captioning would be helpful to his or herunderstanding of the conversation, the captioning service can be addedwhile the call is in progress. To add the captioning service, theassisted user simply has the PICT device dial the relay over the secondtelephone line. The voice of the hearing user is then transmitted overthe second telephone line to the relay. The relay converts the voice totext and the text stream created by the relay returns to the assisteduser, also over the second telephone line. Note that in thisarrangement, as indicated in FIG. 5, voice and text are transmitted inopposite directions over a single telephone line between the PICT deviceand the relay, voice going to the relay and text returning on the sameline. This arrangement of voice and text is the same as used for thepersonal interpreter.

One device intended to implement the two-line captioned telephoneapproach is illustrated in FIG. 6. The PICT device 80 of FIG. 6 is ananalog device connected in series between the telephone line 64 to thehearing user and the telephone 70 of the assisted user. As seen in FIG.6, the telephone line 64 is connected directly through the PICT device80 to the telephone 70. However, that telephone connection is alsoconnected through a transformer 82 to a 2-wire to 4-wire converter 84.Connected to the output of the converter 84 is a low pass filter 86, theoutput of which is the input to an amplifier 88. The output of theamplifier 88 is connected to a 2-wire to 4-wire converter 89 whichconnects, in turn, to the telephone line 78 which connects to the relay(not shown here). The input signals from the telephone line 78 areconnected to a hi-pass filter 90, the output of which passes through areceive modem 92 to a microcontroller 94. The microcontroller operatesthe display 96. The microcontroller also includes the capability togenerate DTMF dialing tones and impress those tones on the telephoneline 78 when it is time to dial the relay. Thus here the single PICTdevice 80 connects to two telephone lines. A call can be initiated orreceived by the assisted user in the same manner as with a conventionaltelephone, using only the first telephone line 64. When the user wishesto invoke the captioning service, the assisted user presses a button,97,on the PICT device 80 that causes the device to automatically dial therelay on the second telephone line 78. The PICT device automaticallytransmits the voice signals on the first telephone line to the secondtelephone line (through the low-pass filter 86) for transmission to therelay. At the relay, the voice is converted into text and a text streamis sent back down the telephone connection 78 to the PICT device. ThePICT device filters the text data, at the hi-pass filter 90, andtransmits it to the microcontroller 94 for display to the user. None ofthe text data, and no digital carrier signals, are heard by either thehearing user or the assisted user. Normally the hearing user does noteven need to be aware that captioning is being used on the call.

This example assumes that the digital carrier signals for the textmessage are carried on high frequencies, and the low pass and high passfilters are used to separate voice from text. While this filtering canbe implemented as an analog filter, in many telephonic systems today,digital forms of communication are used. Using a digital communicationprotocol between the relay and the PICT device, communication is in theform of digital data packets of either text or digitized voice. In thatevent, the logical filtering simply consists of not creating anyacoustic noise from the packets designated as carrying text.

Several physical versions of the PICT device are contemplated. In theversion illustrated in FIG. 5, the PICT device is a stand-aloneappliance inserted in series between the telephone of the assisted userand the telephone jack on the wall of the user's home. It may also beconvenient for the user to package the PICT device as a single two-lineunit intended to provide both conventional telephone service, as well ascaptioned telephone services. The device could look like a normaltelephone, but have a display. The captioned telephone feature would beactivated by a button or key, which would cause the unit to dial apre-selected relay and set up a captioning session, whether or not acall on the other line is yet in progress. The voice signals incoming onthe first telephone line from the hearing user would be transferredelectronically to the telephone line to the relay. Text signals would betransmitted on that same telephone line back from the relay to the PICTdevice, which would display that text for the assisted user. Since there-voicing relay operates with only a very slight time delay, the textdisplay for the assisted user would trail the hearing person's voiceslightly, but the text would still assist the user in comprehending theconversation.

It is a desirable feature of the two-line captioned telephonearrangement that the use of the captioning service is transparent to thehearing user. The hearing user would dial to connect to the assisteduser as with any other telephone user. The assisted user invokes thecaptioned telephone service without the need to involve the hearing userat all. The service can be used equally well and transparently for bothincoming and outgoing calls.

It is also envisioned that the captioned telephone service can beimplemented in a way that uses three telephone lines. One line is forvoice communications with the hearing user. The second line is directlyconnected to the first line so that the voice of the hearing user istransmitted to the relay. The third line is a connection to transmit thetext stream from the relay to the station of the assisted user.

In the implementation of a relay providing captioned telephone service,the relay receives the voice of the hearing user and transmits both adigital text message stream and the voice of the hearing user over atelephone connection to the station of the assisted user. It is to beunderstood, however, that a conventional telephone single lineconnection is only one example of a telephonic connection that can beused in this arrangement. Digital wireless connection, or PCSconnection, or even internet protocol (IP), wired or wireless connectioncan be used to connect the relay to the assisted user, so long as theconnection in capable of trasmitting voice to that user.

Thus the term telephone line as used in this specification is intendednot only to apply to a traditional land-line two-wire telephone line,but also to all equivalents that offer similar functionality. Each ofthe telephone lines could be, for example, a portion of the bandwidth ofan ISDN or DSL service. The telephone line could be an analog or digitalcellular telephone link or a PCS connection. The PICT device could alsobe connected to the internet communication in IP, and in that event thetwo telephone lines would simply be simultaneous digital data exchangewith two remote locations.

It is to be understood that the present invention is not limited to theparticular illustrations and embodiments disclosed above, but embracesall such modified forms thereof as come within the scope of thefollowing claims.

1. A captioned telephone device for providing captioned telephoneservice to an assisted user communicating with a hearing user through atelephone connection using a relay having speech to text translationcapability, the hearing user speaking words in voice, the devicecomprising: a microphone; a speaker; a visually readable display;circuitry to support connection to two telephone lines; and amicroprocessor programmed to operate the device to: receive a telephonecall over a first telephone line directly between the assisted user andthe hearing user; initiate a telephone connection over a secondtelephone line to the relay; transmit the voice of the hearing user overthe second telephone line to the relay so that the relay can convertingthe words spoken by the hearing user into text and transmit the textcreated by the relay back to the device over the second telephone line;and display the text on the display within sight of the assisted usersuch that captioning of the communication session is provided to theassisted user.